Repository navigation
Conversation
certify.sh attaches a certification note on refs/notes/certification and pushes it; pick-release-candidate.sh selects the newest certified commit the release train has not released yet. test_release_train.sh covers note creation, idempotency, and every selection rule.
Certifies main HEAD with the release-validated gate (testDebug/testAsan, regular and slow, gtests), fuzz, and the dd-trace-integration cells read from the commit's GitLab pipeline. Green runs attach a certification note; failures notify Slack and file a PROF ticket.
The release train picks the newest unreleased certified commit and dispatches release-validated.yml pinned to it. release-validated.yml and release.sh accept a commit on main as the release ref; the manual flow is unchanged. ci.yml runs the new hermetic tests.
update_assets.yml runs bump-ddtrace.sh once the release artifacts are live on Maven Central; the script bumps the ddprof version in libs.versions.toml and the lockfile and opens a draft PR.
Fail the train when the dispatched release fails (an ::error:: annotation does not fail the step) and default dry_run on scheduled runs, which left the dispatch with an empty boolean input. Pick the newest certified commit by history position instead of committer date, fail loudly on notes-fetch errors, allow non-fast-forward notes updates, and let the bump job treat the missing-token exit as a documented skip.
Select the tip-most certified commit with awk over rev-list --topo-order: the grep|head pipeline aborted under set -euo pipefail once more than one certified commit existed (grep exit 1 on no match, SIGPIPE on many). Distinguish a missing notes ref (ls-remote exit 2) from a transport failure; URL-encode the GitLab ref filter with a sha-only fallback; disambiguate the dispatched-run lookup by dispatch time and raise the retry budget; report the train summary on failure; push the bump branch before recreating it.
Absorb runner/GitHub clock skew in the dispatched-run filter with a 5-minute margin; fail loudly when rev-list fails instead of reporting no candidate; guard the bump re-push after a branch recreate.
The title timestamp window could misattribute a concurrent manual dispatch of the same candidate; request_id is unique to this train's dispatch and appears in the run title.
The run title ends with a closing bracket after the request_id, so endswith never matched.
last | .databaseId on an empty array prints the literal null, which broke the retry loop on its first attempt.
Prevents a longer concurrent run id with the current one as a prefix from matching.
|
@DataDog review |
CI's shellcheck fails SC2015 on it.
There was a problem hiding this comment.
Certification is rejected by workflow validation, and a missing job-output mapping breaks automated and manual releases. Additional defects affect GitLab matrix evaluation, release-candidate safeguards, dependency updates, and failure notifications.
🤖 Bits Code Review · Commit b618429
Map base_branch and target_sha into the validate-inputs job outputs; pin the release commit through source_sha with ref=main (the dispatch API takes a branch ref, not a SHA); run version computation at the release commit; move secrets checks into scripts; aggregate reusable-workflow job conclusions; configure the git identity for the certification note; keep script diagnostics on stderr; gate candidate selection on mainline tags only and apply the same rule to manual overrides; use the /search/jql endpoint with an ADF comment body; refuse a ddprof pin downgrade; compose the Slack message from explicit outputs.
CI Test ResultsRun: #45073230513 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-11-22 22:04:43 UTC |
Pass the release commit to test_workflow.yml through a new ref input so the pre-release gate tests the certified commit, not main HEAD. Pass source_sha through env (dispatcher-controlled input must not be interpolated into the shell program). Require override candidates to be reachable from origin/main.
Remove the duplicated ref key in the musl-aarch64 checkout; record and pass the tested commit (inputs.ref) in test metadata and the aarch64 docker script instead of the dispatch ref.
|
@DataDog review |
There was a problem hiding this comment.
Certification cannot recognize the existing GitLab matrix jobs, preventing new certified release candidates. The release and dependency-bump paths also contain maintenance-commit validation and Git authentication defects.
🤖 Bits Code Review · Commit 826797b
Match GitLab matrix-expanded integration cells by prefix and aggregate cell statuses; paginate the job lookup; validate pinned patch/retag commits against the release branch; include the run attempt in the dispatch request_id; wire the token into git credentials for the bump push; abort the bump on a PR-lookup failure.
gh api rejects --slurp with --jq: paginate per page and aggregate conclusions in awk. Follow GitLab job pagination for matrix-expanded pipelines; classify terminal non-success cells (failed, canceled, skipped, manual) as failed instead of spinning on them; match matrix cells by name or its ': [' suffix. Bump PR lookup falls back to empty instead of the literal null. Distinguish merge-base errors from not-an-ancestor. Serialize release runs per candidate with a release-validated concurrency group.
|
❌ 38 passed, 0 failed, 2 without a result out of 40 expected configurations Test Matrix
No result producedThese configurations wrote no validation log, so their outcome is unknown and
Links
|
String-concatenating page arrays produced invalid JSON, and hoisting the fetch out of the retry loop froze cell statuses on the first attempt.
Suffix cache_java artifact names per caller so the four certification suite invocations in one run do not collide on sdkman-installation-*; fail the GitLab job fetch on curl errors instead of masquerading as an empty complete listing; file the PROF ticket even when Slack fails; refuse a release commit that already carries a tag (queued duplicate dispatch after a concurrent release).
The two pre-release test calls share one run and need distinct artifact_suffix values like certification. Notify Slack and File PROF ticket run on failure only, not on cancellation. Pre-seed the GitLab STATUS map so an all-fetch-failures path reports unknown instead of an unbound-variable abort. Avoid the dangling hyphen with an empty artifact_suffix.
|
@DataDog review |
There was a problem hiding this comment.
An accepted source revision can execute shell substitution after the release SSH key is loaded, even during a dry run. Release serialization, retag recovery, and concurrent dependency bumps also need fixes.
🤖 Bits Code Review · Commit d7f22c8
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7f22c8ec6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
A failed certification can finish its reporting job without sending Slack or filing a ticket. The release workflow also blocks retag recovery when publishing failed at the unchanged release commit.
🤖 Bits Code Review · Commit d7f22c8
Mainline releases share one concurrency group: a queued run validates before the previous run's tag exists, so per-SHA grouping lets two dispatches release the same commit. The pinned source is canonicalized to its commit ID so a tag name or revision expression can neither defeat the duplicate-release guard nor reach the release job's checkout command. retag is exempt from the duplicate-release guard: it re-points an existing tag at the same commit by design. The bump job serializes against overlapping manual runs. Notification steps run on !cancelled() (the job is gated on failure() || cancelled(); step-level failure() is false when only a dependency failed). Pipeline selection prefers the push pipeline over same-SHA trigger pipelines that lack the integration jobs.
GitLab pipeline selection sorts ascending: the push pipeline is the oldest for a SHA and sort=desc would push it off the first page behind the trigger pipelines it spawned. The unresolvable-source error keeps the dispatcher's original input. The update-assets workflow serializes per tag so a tag-push run and a manual run for the same version cannot race the asset uploads and the --latest flag.
|
@DataDog review |
There was a problem hiding this comment.
More details
The PR automates nightly certification, weekly minor releases, and dd-trace-java dependency bumps.
All previously reported issues are fixed: source-ref normalization,
retag recovery, certification notifications, GitLab pipeline selection,
and concurrent dependency-bump branch deletion;
no findings remain open.
🤖 Bits Code Review · Commit e5e0337
There was a problem hiding this comment.
The train can publish an older certified commit after a newer manual major release completes while it waits for the shared release lock. The duplicate-tag guard does not enforce release ordering.
🤖 Bits Code Review · Commit e5e0337
…tract A queued mainline dispatch that predates a newer release (tagged while it waited for the concurrency lock) passed the equality-only duplicate guard; the mainline guard now also rejects a release commit that does not contain a v_* tag already on main. Certification and release-train Slack payloads carry the scenarios and failed_run_url fields that the nightly.yml receiver consumes, alongside text.
One pass over the release tags now runs both guards: the duplicate-release equality check and the staleness check against the release branch's tags (main for major/minor, the release branch for patch), so a queued dispatch cannot publish a commit older than the newest tag on its target branch. The release train's skip notification stays plain text: the nightly receiver treats the scenarios/failed_run_url fields as a failure report.
|
@DataDog review |
There was a problem hiding this comment.
The PR automates certification, weekly releases, and dd-trace-java bumps;
prior fixes cover source pinning, retag recovery, notifications,
pipeline selection, and bump concurrency.
The new release-position guard rejects major/minor dispatches
from tags by resolving nonexistent remote branches.
🤖 Bits Code Review · Commit 689a7e7
There was a problem hiding this comment.
A stale-branch manual dispatch can select an older commit for release after a newer mainline release. A local reproduction confirmed that validation accepts this case; the suggested replacement rejects it while preserving maintenance-branch handling.
🤖 Bits Code Review · Commit 689a7e7
zhengyu123
left a comment
There was a problem hiding this comment.
Review findings on 689a7e7. Each was checked against the code at this commit. Items 1–3 can break a release or certification; the rest are smaller.
- Retag can rewind a release branch (
release-validated.yml). - Older-line patch releases end with a red
bump-dd-trace-javajob (update_assets.yml). fetch_all_jobsloses jobs past 100 (certification.yml).bump-ddtrace.shfails when the lockfile is absent.- Smaller items:
RELEASE_BASE_BRANCHnot exported, runbook fallback, job timeout, test coverage, duplicated comment.
|
🐶 babysit-pr is watching this PR — CI breakages and mechanical findings are fixed and pushed automatically; design decisions are always brought to you. This run
✅ stopped 20:51 UTC · blocked on human action: 16 confirmed findings in human-gated .github/ CI scope await manual fixes; test matrix green on 689a7e7; bits-review FAIL + codeowner approval pending Stop summaryCommits pushed: none — every finding landed in CI configuration/scripts or docs outside the babysitter's edit scope (sensitive .github/ paths also gated on pending @DataDog/profiling-java approval), so no fix-push cycle was run. Findings flagged for manual application (all confirmed against the code):
Threads left for the author: none unanswered — all 16 new trusted threads this run carry exactly one babysit reply; open threads from earlier rounds were already answered before this run. bits_triggered heads: d653ef9, e5e0337, 689a7e7 (no new head this run; the FAIL re-review of 689a7e7 was an external re-trigger). |
|
@DataDog review |
There was a problem hiding this comment.
Pinned retagging can rewind a maintenance branch. The release-age guard also checks the dispatch branch rather than main, allowing a mainline release to publish older code.
🤖 Bits Code Review · Commit 689a7e7
| git config --global user.email "java-profiler@datadoghq.com" | ||
| git config --global user.name "Datadog Java Profiler" | ||
| git checkout -B "$GITHUB_REF_NAME" "$GITHUB_SHA" | ||
| git checkout -B "${{ needs.validate-inputs.outputs.base_branch }}" "${{ needs.validate-inputs.outputs.target_sha }}" |
There was a problem hiding this comment.
Reject retag targets behind the release branch tip
For an unpublished release, retag validation accepts a source_sha behind the maintenance branch tip. This checkout resets the local branch to that historical commit, and release.sh then force-pushes it. Because the fresh clone's tracking ref matches the current remote tip, --force-with-lease permits the rewind, removing subsequent commits from the remote branch. Check the target against the freshly cloned branch tip before resetting it.
| git checkout -B "${{ needs.validate-inputs.outputs.base_branch }}" "${{ needs.validate-inputs.outputs.target_sha }}" | |
| if [ "${{ inputs.release_type }}" == "retag" ] && [ "${{ needs.validate-inputs.outputs.target_sha }}" != "$(git rev-parse "origin/${{ needs.validate-inputs.outputs.base_branch }}")" ]; then | |
| echo "::error::Retag must target the current tip of the release branch" | |
| exit 1 | |
| fi | |
| git checkout -B "${{ needs.validate-inputs.outputs.base_branch }}" "${{ needs.validate-inputs.outputs.target_sha }}" |
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Duplicate of the thread on this line from the earlier review pass: retag validation accepts a source_sha behind the release branch tip, the checkout resets to it, and release.sh's fresh-clone --force-with-lease permits the rewind. Checking the target against the freshly cloned branch tip is queued for manual application (CI config, human-gated).
| exit 1 | ||
| fi | ||
| RC=0 | ||
| git merge-base --is-ancestor "$TAG_COMMIT" "origin/$BRANCH" || RC=$? |
There was a problem hiding this comment.
Check release age against the effective target branch
Major/minor dispatches now accept a source_sha on main regardless of the dispatch branch, but this filter still checks tags against that dispatch branch. Newer mainline tags absent from a stale branch are skipped, allowing the age guard to approve older source—for example, a major release computed as 2.0.0 from before the latest 1.33.0 release. Compare against origin/main for major/minor releases.
| git merge-base --is-ancestor "$TAG_COMMIT" "origin/$BRANCH" || RC=$? | |
| RELEASE_TARGET_BRANCH="main" | |
| if [ "$TYPE" == "patch" ] || [ "$TYPE" == "retag" ]; then | |
| RELEASE_TARGET_BRANCH="$BRANCH" | |
| fi | |
| git merge-base --is-ancestor "$TAG_COMMIT" "origin/$RELEASE_TARGET_BRANCH" || RC=$? |
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Duplicate of the two threads on this line from the earlier Bits pass: the tag-ancestry filter must run against the effective target branch (origin/main for major/minor, origin/$BRANCH for patch/retag), else newer mainline tags are skipped on stale dispatch branches. Queued for manual application (human-gated .github/ scope).
| if [ -z "$jobs_json" ]; then | ||
| jobs_json="$page_json" | ||
| else | ||
| jobs_json=$(jq -cn --argjson a "$jobs_json" --argjson b "$page_json" '$a + $b') |
There was a problem hiding this comment.
Merge GitLab job pages through stdin
Passing accumulated job JSON through --argjson exceeds Linux's per-argument limit once a page grows beyond approximately 128 KiB. A focused reproduction using 100 jobs with 1,500-byte commit messages produced 'Argument list too long', yet the caller accepted an empty listing because the conditional function call suppresses errexit. Required integration jobs then appear missing and certification fails. Merge through stdin and propagate jq failures.
| jobs_json=$(jq -cn --argjson a "$jobs_json" --argjson b "$page_json" '$a + $b') | |
| jobs_json=$(printf '%s\n%s\n' "$jobs_json" "$page_json" | jq -sc 'add') || return 1 |
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Duplicate of the thread on this line from the earlier pass — now with a focused reproduction confirming 'Argument list too long' and the silently accepted empty listing. Merging pages through stdin and propagating the jq failure matches what is queued for manual application.
| echo "ERROR: refusing to downgrade ddprof from ${OLD_VERSION} to ${VERSION}" >&2 | ||
| exit 1 |
There was a problem hiding this comment.
Skip dependency bumps for older maintenance releases
The asset workflow invokes this script for maintenance releases too. If dd-trace-java already pins 1.33.0, publishing 1.32.2 reaches this exit 1, making the workflow fail after successfully publishing the release and assets. Preserving the newer dependency is expected behavior and should return a successful skip, as the already-pinned case does.
| echo "ERROR: refusing to downgrade ddprof from ${OLD_VERSION} to ${VERSION}" >&2 | |
| exit 1 | |
| echo "dd-trace-java already pins newer ddprof ${OLD_VERSION}; skipping ${VERSION}" | |
| exit 0 |
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Duplicate of the thread on update_assets.yml:109 from the earlier pass: the downgrade refusal's exit 1 turns an already-published maintenance release red. Returning a successful skip, like the already-pinned case, is queued for manual application.
The retag path forced the release branch to the dispatch target, so a retag of a non-tip commit rewound the branch and dropped commits; retag now requires the current branch tip. The staleness guard compared tags against the dispatch ref, which breaks for tag-named refs and misses newer mainline tags on a stale branch; it now checks main for major/minor and the dispatch branch for patch/retag. The GitLab job-page merge moved to stdin (--argjson can exceed the per-argument limit and fail silently) and the job timeout now exceeds the combined retry budgets. bump-ddtrace.sh skips (exit 0) when dd-trace-java already pins a newer version, adds the lockfile conditionally, and uses portable sed -i; its flows are covered by new hermetic tests. RELEASE_BASE_BRANCH reaches release.sh through the step env, the doc's older-candidate fallback names a working path, and the duplicated guard comment is gone.
The downgrade skip carries a ::warning:: so it stays visible in the workflow run; the sed backups are removed on the verification-failure paths too; the staleness-guard comment states that only major/minor dispatches can carry a tag-named ref; the missing-lockfile test bumps a version unique to its case so it cannot pass on the earlier case's branch.
There was a problem hiding this comment.
More details
The PR automates nightly certification, weekly minor releases,
and dd-trace-java bumps, with prior fixes for source pinning,
notifications, pipeline selection, retag recovery, and bump concurrency.
The release-history guard now uses the effective target branch,
resolving tag-based dispatch failures; no review findings remain open.
🤖 Bits Code Review · Commit 7128c34
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 4 Pipeline jobs failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 7128c34 | Docs | View more details | Give us feedback! |
What does this PR do?:
Adds a fully automated release train on top of the existing tag-based release flow:
.github/workflows/certification.yml, 04:00 UTC) runs therelease-validated.ymlpre-release gate (testDebug+testAsan, regular and slow, gtests), the fuzz job, and reads thedd-trace-integrationcells from the commit's GitLab pipeline. A green run attaches a git note on the certified commit underrefs/notes/certification; a failed run notifies Slack and files a high-priority PROF ticket (deduped per commit)..github/workflows/release-train.yml, Fri 11:00 UTC) picks the newest certified commit the lastv_*tag does not cover, and dispatchesrelease-validated.ymlpinned to that commit (minor,dry_run=false). It never releases twice from the same certified commit and never releases a commit older than the last release; both rules are enforced inpick-release-candidate.shand re-checked before dispatch. Skips with a Slack note otherwise.update_assets.yml+bump-ddtrace.sh): once the release artifacts are live on Maven Central and the GitHub release is published, a draft PR onDataDog/dd-trace-javabumpsgradle/libs.versions.tomlanddd-java-agent/ddprof-lib/gradle.lockfile(the exact two files a manual bump touches) with labels and a link to the release notes. A human merges it.Both new workflows accept manual
workflow_dispatchruns; the train takes optionalshaanddry_runinputs for controlled trials. Major and patch releases stay manual.release-validated.ymlandrelease.shnow accept a commit onmainas the release ref (pinned dispatch); the manual flow is unchanged.Docs:
doc/building/ReleaseTrain.md(+ index,Versioning.md,utils/README.md,AGENTS.mdpointer).Motivation:
Releases and dd-trace-java bumps are operator-driven today; the release gate re-runs the same suite a nightly already ran. The train makes the weekly minor release hands-off while keeping the published-artifact guarantee: only commits that passed the full certification suite can be released. PROF-16810 records the design decisions.
Additional Notes:
Review: agentic Sphinx review (16 range reviews across the fix rounds) plus eight Bits trusted-bot review rounds and one human review round (12 + 6 + 4 + 4 + 5 + 2 + 4 + 12 findings, all addressed). The initial Sphinx review found 19 findings (3 HIGH: a failed dispatched release did not fail the train, and scheduled runs dispatched
dry_run=with an empty boolean); the Bits round caught a further 12 (missingbase_branch/target_shajob outputs,source_shareachable as shell input, dispatch-ref pinning semantics, retired Jira search endpoint, mainline-vs-maintenance tag gating); every fix was re-reviewed and its acceptance brief verified; the final fix rounds found 0 HIGH/MEDIUM findings.Bits rounds 3–5 caught: artifact-name collisions when one run invokes the test workflow repeatedly (now disambiguated by an
artifact_suffixthreaded through to the cache and failure artifacts), a GitLab page-fetch failure reading as a complete listing, a queued duplicate dispatch double-releasing a certified commit, the pinned source SHA reaching the checkout command uncanonicalized (tag names and revision expressions), manual mainline releases racing the train's concurrency group,retagrecovery blocked by the new duplicate-release guard, an unserialized dependency-bump race, notification steps skipping when only a dependency job failed, and pipeline selection returning a trigger pipeline without the integration jobs. A follow-up Sphinx review of those fixes verified all six sub-claims and caught two regressions (page order undersort=deschides the push pipeline; step-levelfailure()skips notifications), both fixed. One open LOW:pick-release-candidate.shleaks itsmktempfiles if it aborts between creation and cleanup (pre-existing pattern, ephemeral CI runner). Two late Sphinx fix-review sessions wedged before delivering their sealed reports (invalid_statein the verification phase); their completed rounds found only LOW/MEDIUM items, and one undelivered MEDIUM from the final certification-workflow round remains unexamined — flagged for the next reviewer.Review trail: initial
1d5a5519..bfab1a00d, then per-fix ranges through1a38bf2a..d653ef961(the review marker lists the ranges). Human review is recommended before merging: this change automates the release path and adds consumers of four new repo secrets (GITLAB_API_TOKEN,ATLASSIAN_BASE_URL/ATLASSIAN_USER_EMAIL/ATLASSIAN_API_TOKEN,DDTRACE_GH_TOKEN) that must be provisioned before the automation is live; certification cannot certify without the GitLab token by design.How to test the change?:
.github/scripts/tests/test_release_train.sh— hermetic (23 assertions): note creation, idempotency, and every candidate-selection rule including the no-double-release and no-older-than-last-release guards, multi-note tip selection..github/scripts/tests/test_release_automation.shstill passes (44 groups) against the modifiedrelease.sh; both run inci.ymlwithbash -nandshellcheck.qualityCheckspasses, including the doc index.gh workflow run certification.ymlonmaincertifies HEAD;gh workflow run release-train.yml -f dry_run=truepicks the certified commit and dispatches a dry-run release (no tag, no GitLab trigger).For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.